Test failure analysis: real bug, already fixed upstream, blocked on an MSBuild version lag

The CI test failures on this PR (e.g. Microsoft.TemplateEngine.Authoring.Tasks.IntegrationTests LocalizeTemplateTests / ValidateTemplatesTests, plus dotnet-watch.Tests) are real, not flaky, and are a direct consequence of enabling the MSBuild server by default in MSBuildForwardingAppWithoutLogging.

Root cause

With MSBUILDUSESERVER=1, TerminalLogger auto-detection runs in the persistent server node and inspects the node's stdout instead of the client's. When the client's stdout is redirected (as it is when tests capture dotnet build output), the server still emits terminal-logger ANSI output. So assertions like HaveStdOutContaining("Build FAILED.") fail because the output is the TL-formatted failed with 1 error(s) instead of the classic console-logger Build FAILED..

This is exactly dotnet/msbuild#14077 ("Fix TerminalLogger auto-detection under MSBuild Server", fixes dotnet/msbuild#13940), merged 2026-06-17, which explicitly calls out the Windows > file.txt (redirected) case.

Reproduction (pure CLI, no test harness)

Using the SDK built from this branch, a redirected dotnet build of a failing project:

Deterministic, reproduces without any test framework. Verified end-to-end that the daily .NET 11 SDK (11.0.100-preview.7.26360.102, MSBuild 18.10.0-1.26360.102, post-#14077) makes server-on behave identically to server-off (plain output, no ANSI) — so the fix resolves it.

Why CI still fails (the blocker)

The SDK builds against / redistributes MSBuild via MicrosoftBuildVersion, which resolves from eng/Version.Details.props:

MicrosoftBuildPackageVersion = 18.9.0-preview-26311-113

That MSBuild is source-dated 2026-06-11 — six days before the fix. Note this is out of sync with eng/Version.Details.xml, which was VMR-source-updated to Microsoft.Build 18.9.0-preview-26325-102 (2026-06-25, which does contain #14077). Both files are identical on origin/main, so this is not a stale-branch issue on my side — main's redistributed MSBuild simply predates the fix.

What would unblock it

The SDK needs a post-#14077 MSBuild (>= 18.9.0-preview-26325-102; confirmed the fix is present in 26325 and daily 26360). Either wait for the dependency flow that bumps MicrosoftBuildPackageVersion in eng/Version.Details.props, or bump it manually as part of this PR.

Happy to do whichever the maintainers prefer.

Update: MSBuild bump to 26325 fixed the Windows TL failures; residual failures are a separate Unix server bug

Pushed 31ecd76 bumping the redistributed MSBuild to 18.9.0-preview-26325-102 (which contains dotnet/msbuild#14077). Result on the next CI run (build 1503449): failed work items went 7 → 3, and all Windows legs are now green (TestBuild: windows 214/214, FullFramework: windows 152/152) — confirming the TerminalLogger-under-server fix resolves the Windows failures.

Remaining 3 failures (all Unix)

The first two are the same root cause, and it is not the TerminalLogger bug: with the server enabled on Unix, an MSBuild invocation that succeeds (0 errors, Build succeeded.) intermittently returns exit code 1. It hit a different test each run (previously CanRunTask, now CanRunValidateTask_OnInfo), which points to an intermittent exit-code race in the server node on Unix rather than a deterministic failure. I could not find an existing MSBuild fix for it, and it reproduces only on Unix (I'm on Windows, so I can't repro/fix it locally).

Net

• Windows: green with MSBuild ≥ 26325.
• Unix: blocked on a separate, still-unfixed MSBuild-server "exit code 1 on successful build" issue that enabling the server by default surfaces.

This looks like it needs an upstream MSBuild fix (or confirmation that these are acceptable/known-flaky) before the server can be defaulted on for Unix. Flagging for maintainer guidance.

Local repro investigation of the residual Unix failures (WSL Ubuntu 24.04)

To determine whether a newer MSBuild would fix the two Unix "exit code 1 on a successful build" failures, I tried to reproduce them locally with the exact SDK CI uses (11.0.100-preview.7.26325.102, MSBuild 18.9.0-preview-26325-102) and the daily (26360, MSBuild 18.10.0):

• Sequential dotnet build of a message-logging succeeding project under the server: 0 failures.
• Concurrent (8×6) dotnet build under the shared server: 0 failures.
• Faithful replication of the dotnet-watch invocation (dotnet msbuild /restore /t:GenerateWatchList /p:DesignTimeBuild=true with the real DotNetWatch.targets + DotNetWatchTasks.dll, 2-project reference graph), sequential and concurrent, on 26325: 0 failures in 130+ invocations.

So the exit-code race is rare and timing/parallelism-dependent — it did not reproduce once locally, yet CI hits ~1 test per run (a different test each time). I therefore can't confirm locally whether bumping MSBuild further (e.g. to 18.10.0-1.26360.102) fixes it, and a 18.9→18.10 skew against the rest of the 26325 dependency set carries its own risk, so I did not make that change.

Summary for maintainers

• The 26325 MSBuild bump fixes the deterministic Windows TerminalLogger failures (Windows now green).
• The remaining Unix failures are a rare, intermittent MSBuild-server "exit code 1 on success" race (CanRunValidateTask_OnInfo, dotnet-watch ProjectReferences_OneLevel), plus one unrelated flaky unit test (ItReturnsNullWhenCommandNameDoesNotExistInProjectTools, no MSBuild involved).
• These look like an upstream MSBuild-server correctness issue surfaced by defaulting the server on, rather than anything fixable in the SDK. Guidance welcome on whether to (a) pursue an upstream fix, (b) treat as known-flaky, or (c) gate the default differently.

Update: rebased on main + bumped MSBuild to 26360 (includes the exit-code fix #14251)

Rebased onto current main and confirmed main's MSBuild (18.10.0-preview-26359-117, VMR 2026-07-09) does not yet contain the exit-code race fix — dotnet/msbuild#14251 (fixes #14172) back-flowed into the VMR after that build.

Issue #14172 is an exact match for the two residual Unix failures: on Linux with the MSBuild Server, a successful build intermittently exits 1 with Build succeeded. dropped, and it only reproduces under CPU load (which is why my earlier idle WSL stress runs found nothing, and why it shows up on busy CI Helix agents). The issue even names SDK 26325.

Changes in this push

• eng/Version.Details.props: bump Microsoft.Build / Microsoft.Build.Localization / Microsoft.Build.TaskAuthoring.Analyzer from 18.10.0-preview-26359-117 → 18.10.0-1.26360.102 (VMR commit c98e79e9, 2026-07-10). I verified via source that this build contains [release/5.0.1xx] Update dependencies from dotnet/msbuild #14251 (ProcessPacketsUntilBuildFinished + DrainPacketQueue + the [master] Update dependencies from microsoft/vstest #14172 regression test hook).
• Added IL2026 suppressions at the 6 CLI MSBuild-object-model call sites, because 18.10 adds [RequiresUnreferencedCode] to ProjectInstance.Build/BuildManager (matching the existing IL3050 suppression pattern).

Local Debug build is clean and the previously-failing Microsoft.TemplateEngine.Authoring.Tasks.IntegrationTests pass (6/6) with the server on. Pushed to see if CI (especially the Unix Helix legs) goes green.

Note: the 18.10.0-1.26360.102 pin is a manual bring-forward to get #14251 for testing; it should be reconciled with the normal VMR dependency flow (i.e. dropped once main flows a VMR build that already contains #14251).

there seems to be no problem after this #55245

codeflow this was dependending on to fix msbuild bugs merged so the tests pass

I noticed a perf improvement in dotnet restore time.

diff where improvement occurred: dotnet/dotnet@cb8306a...e78e1a0

Any chance this PR could be the reason?

If you are interested here is a link to the dashboard : dashboard

@Nigusu-Allehu agent inspected the traces and the scenario source and yes it's explainable by this change.
However I'm not sure if all of your scenario definitions are correct now. The improved ones are measuring warm restore (a case when MSBuild was already running it's daemon on the machine):
When the main "MSBuild server" is reused it saves startup, JIT and reconnect to "worker node" time 🎉.
But the "Cold" scenario is also valid and interesting. If you want to define that, there needs to be in the setup "dotnet build-server shutdown" call before the dotnet restore or all dotnet.exe/msbuild.exe processes need to be killed.

@Nigusu-Allehu agent inspected the traces and the scenario source and yes it's explainable by this change. However I'm not sure if all of your scenario definitions are correct now. The improved ones are measuring warm restore (a case when MSBuild was already running it's daemon on the machine): When the main "MSBuild server" is reused it saves startup, JIT and reconnect to "worker node" time 🎉. But the "Cold" scenario is also valid and interesting. If you want to define that, there needs to be in the setup "dotnet build-server shutdown" call before the dotnet restore or all dotnet.exe/msbuild.exe processes need to be killed.

Good point. The current implementation only waits for the warmup restore's client process to finish. it doesn't kill the MSBuild daemons.

Rather than just fixing this, maybe we add it as its own scenario: add a warm variant, and keep a cold-process variant that runs  dotnet build-server shutdown  + kills lingering  msbuild* / dotnet*  before the measured restore. The delta between them would tell us exactly how much the daemon processes are helping, and guard against regressions in reuse effectiveness.